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Results-Based Auditing 


1 INTRODUCTION 


01. The Office is committed to improving the quality of our work to make a difference for 
Canadians. As part of this improvement we are shifting emphasis from systems based to a 
results based audit process. We are adopting this more results-based or "results-oriented" 


approach in our value-for-money audits of departments and agencies and in special 
examinations of Crown corporations. 


02. For those of us serving in functional and/or advisory capacities, we have not yet seen 
a wide translation of the concept into practice. We are still very much at the beginning of 
what may be a lengthy, yet fascinating joumey. 


.03 The primary reason that practice has not advanced as quickly as expected appears to 
be the difference in interpretation of, or uncertainty about, the meaning of results-based — 
auditing. As well, it demonstrates that such a major change in focus and approach is difficult. 


.04 This paper is intended to address a number of questions raised about results-based 
auditing. It is clearly not intended to be definitive but, hopefully, it will prompt more discussion 
and debate on the subject and will encourage greater adoption of the results-based approach 


in Our audit work. If you have any questions on this subject, the authors would welcome 
hearing from you. 


2. SHIFTING THE EMPHASIS FROM SYSTEMS TO RESULTS 


.05 Management renewal initiatives in the public and private sector have argued for the 
importance of emphasizing the results to be achieved by programs and services rather than 
systems and processes per se. Maintaining essential systems and controls continues to be a 
responsibility of management. These systems, however, should serve clear management 
objectives and not be ends in themselves. There has been a considerable number of 
references in recent Office initiatives and annual Reports to the importance of government 
entities focusing on results and achieving intended objectives (See Appendix 2.) 


.06 As auditors we are part of the cail to focus more on the objectives and results of the 
entities and programs we audit. We should also reflect on the processes we follow in our 
special examination and value-for-money work and adopt more of a results approach. For 
many, this will constitute a fairly substantial change in the way in which we approach value- 


for-money audits. This increased focus on "results" is incorporated in the "Special 
Examination Audit Guide" just released. 


07 This shift would not change the requirement to do compliance or authority work 
(however, compliance, as well as “internal controls", are subjects of separate papers). It 
should be understood that "due regard" and a "results approach" are compatible concepts. 


Establishing the results achieved, demonstrating due regard for the "three Es" fulfils an 
essential part of our mandate. 


3. WHAT DOES RESULTS-BASED AUDIT APPROACH MEAN? 


08 — The terminology we use is often replete with process-laden terms (see Figure 1). For 
example, if the subject matter we are describing deals with "how’, generally we are In a 
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Results-Based Auditing 


process mode. Likewise, if we are describing what is being done, without also considering 
what Is being achieved, we are in a process mode. The listing of terms in Figure 1 can be 


expanded, but it is a start to heip us differentiate between the emphases or focuses of the two 
approaches. 


FIGURE 1: DIFFERENTIATION 
Process/systems-based focusses on || Results-based focusses on 


so what? 
inputs/outputs/outcomes/impacts 
ends/consequences 

what is being achieved 


how? 
functions 

means 

what is being done 


09 The Glossary in the Comprehensive Auditing Manual (CAM) gives the following 
descriptions of the two approaches: 


Systems-based auditing. In systems-based auditing, the nature, extent and timing of 
detailed audit procedures are based on the organized study and assessment of the 
significant systems and controls in the audited organization. This study and 
assessment, carried out in the initial phase of the audit, determines which systems are 
important to the management of the audited organization and determines the extent to 
which testing during the examination phase of the audit is required to assess the 
reliability of controls and of reports to Parliament. 


The systems-based approach is designed to enable the auditor to concentrate the 
audit effort in areas where performance is not adequately monitored and disclosed and 
where controls either do not exist or are not operating properly. At the conclusion of 
the examination, the auditor should be in a position to point out the strengths and 
weaknesses of the key systems and related controls and to indicate causes and 
effects of the weakness and to develop conclusions and recommendations. 


Results-based auditing. |n results-based auditing, emphasis is placed on identifying 
the critical results or products to be achieved by a program or a process. These 
results may take the form of outputs produced in relation to those intended, products 
delivered, or immediate goals that must be achieved for the program or process to be 
a success. Once these critical results have been identified, the audit examines them 
to identify weaknesses. The audit may assess whether the products or results 
produced are those that are supposed to be produced according to the program 
mandate. Alternatively, the audit may assess whether a program or process is 
producing products to appropriate standards of quality and timeliness, and in an 
economical and efficient way. The purpose of a results-based audit is to identify 
weaknesses in the results of a program or process and to trace these weaknesses 
back to the set of procedures and activities that may be their cause. 


10 The definition of "systems-based" auditing was established at the time when the 
Comprehensive Auditing Manual was first developed in 1981. The definition of results-based 
auditing was added when the Manual was revised in December 1984. 
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Results-Based Auditing 


1a Whether a "systems based" or a “results based" approach is used, significantly 
changes the starting point of the audit. A focus on results needs to be part of the early 
audit planning and key decision points throughout the audit process. Indeed, the audit 
has to have clear audit objectives. The degree to which a results-based approach is to be 
adopted will usually be evidenced by the terminology used in the audit objectives. 


-12 The auditor may need to spend extra time with the management of the audit entity in a 
results-based approach to obtain a common understanding of key results that the entity is 

trying to achieve. The statement of objectives is often unclear. The auditor will need to reach 
an agreement with management on what the objectives are and the results to be achieved. If 


agreement cannot be reached, the lack of clarity of objectives should be considered for 
reporting. 


ay ES Further, as a practical consideration, auditors should take care to be "reasonable" 
regarding expectations on results. For example, when many organizational changes are 
taking place, we should avoid excessively high expectations of immediate results. A 
reasonable time period for proper results (and measurements thereof) must be allowed. Also, 
auditors must ascertain whether good results were due to the use of excessive resources 


(trade-off between results and efficiency)? Care will have to be taken when forming 
judgments on "reasonableness" of results. 


14 A proper balance is needed in the application of the approach. A commitment to 


results-based auditing implies a continuous questioning of "so what" in planning, 
scoping implementing and reporting. 


4. WHAT DOES RESULTS-BASED AUDIT APPROACH NOT MEAN? 


“19 Discussions surrounding the meaning and intended use of the results-based approach 
has resulted in some misunderstandings or myths about the use of the concept. 
Some of the more frequent myths are set out below. 


16 Itis:an "either/or" situation. 


Not so! Most auditors agree that we have been overly focussed on systems, practices 
and processes without giving adequate attention to what they are achieving. lt is nota 
case of either a "systems" approach or a "results" approach, but rather making sure 
that adequate attention is given to results. 


A balance needs to be struck. An emphasis on results should not mean that auditors 


do not examine other relevant matters (e.g., systems and practices that do not 
contnbute to results). 


Because of the legislative wording relating to special examinations which refers to 
systems and practices, there will always have to be some awareness and 
consideration of systems and practices even with a results-based approach. 


There is a big difference, however, between understanding key systems and practices 
as part of the planning phase and focussing the whole special examination or value- 
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for-money audit on key systems and practices. We are trying to de-emphasize, even 
discourage, the latter practice. 


Special examinations, by their nature and legislative wording, are process audits. 


No. During the first cycle of special examinations, a lot of attention was placed on 


‘systems" and "practices", terms used often in the legislation. However, the Special 
Examination Audit Guide explains that: 


"Systems are only means to ends -- they need to be judged by their ability to fulfil 
those ends (i.e., results). The reference in the FAA to systems and practices and 
frequent reference in this audit guide to "statutory control objectives" may lead some to 
consider that special examinations are process-oriented audits. This is not the case, 
for the special examination opinion is on whether systems and practices provide 


reasonable assurance that results are being achieved (i.e., operations are carried out 
effectively, etc.)." 


A greater emphasis on results means we are going to be auditing "policy". 


Not necessarily so. In fact auditors should exercise great caution if they feel they are 
potentially entering a political policy debate. This is particularly important for value-for- 
money audits examining the achievement of overall program objectives - only one part 
of results-based auditing as shown in Figure 2. 


For special examinations, a results-based approach does not mean that there is a 
switch to commenting on the merits of government policies. In fact, under Part X of 
the FAA, we are specifically precluded from doing this. 


There is a major difference between examining whether established policies are being 
implemented and objectives are being achieved and questioning the nature or ments of 
those policies/objectives. Our focus is on the former, not the latter. 


Our reports will become highly statistical. 


While having more quantitative information in our reports is being encouraged and is 
consistent with a results-based approach, it does not necessarily mean a switch to 
highly statistical reporting. Any quantitative information should be relevant and 
supportive of the conclusion. Numbers for the sake of numbers are not helpful - and 
could be harmful! More quantitative information should mean more reporting on 
performance, comparing actual results to expected results, comparable benchmarks, 
etc. While much of this performance information is (or should be) produced by the 
entity, this does not preclude generating our own information and analyses relative to 
performance. Separate guidance on quantitative methods will be forthcoming. 
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.23 


We wiil be doing program evaluations. 


Program and policy evaluations are carried out by government to answer questions 
about whether objectives have been achieved, whether there is still a rationale for the 
program and whether there are any cost-effective alternatives. In the Canadian system 
we expect government to carry out these evaluations, often with encouragement or 
prodding on the part of the auditor. We put a considerable amount of audit effort into 
demonstrating the importance of evaluations and demonstrating the importance of the 
kind of information that an evaluation can provide, but only in exceptional circumstance 


would we undertake them ourselves. (Auditors should seek functional guidance in this 
area.) 


All “results audits" require functional guidance from the Results Measurement 
Audit Group. 


Not so! The Group, along with the Crown Corporation Group for work in Crown 
corporations, is directed in part to supporting the shift of emphasis to results- based 
auditing by providing assistance. The Results Group also conducts results based 
audits. Members of the results measurement team are available for assistance and 
should be consulted when Principals and/or AAGs are planning their audits for 
efficiency and effectiveness issues as described in CAM. 


A results-based approach has to cost more. 


This is not so. Devoting more time to understanding the industry where one exists, to 
comparative analysis, (often in other jurisdictions) to discussions with management on 
objectives and results are all seen as a change of emphasis - not add ons. Therefore, 
less time is expected on understanding and auditing systems and processes. That 
Savings in time should allow for the increased focus on "results". 


There is nothing in this paper that is in contradiction with the Office's concern about 
audit efficiency. It is the view of the authors that a results-based audit will not only be 
more relevant but may be carried out at a lower cost. We are confident that auditors 
will support this challenge and adopt the results based approach at the same, or lower, 
cost than a systems based approach. 

THE CHANGE IN SPECIAL EXAMINATIONS 

Reasons for change 


The Special Examination Audit Guide cites the following reasons for adopting a results 


onentation. 


Focussing on the results that the corporation is expected to achieve will point the 
examiner to deficiencies that directly affect results. 


The results orientation is likely (but not necessarily) less costly, since only those 
systems and practices essential to achieving results are examined in depth. 
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Results-Based Auditing 


The results approach increases our credibility with the board and senior management 
because it focusses on matters that have a direct impact on the corporation's success. 


ltis easier to appreciate the significance of audit results if we relate them to the 
achievement of desired results and, by extension, on the statutory control objectives. 


-24 Further, the rate of change in Crown corporations is very rapid. Board members and 
senior executives change; operations change; and circumstances change (e.g., technological 


advancements). While corporate mandates and objectives are also subject to change over 
time, they tend to be more durable. 


-25 That is, to focus primarily on systems and practices is to direct the special examination 


on the very things most likely to be in a state of change - most of these things are simply 
“means” to an end. 


.26 By focussing our audit efforts on results achieved in relation to objectives we avoid 
getting caught up in the current means management is using to meet its objectives, and look 
at whether results are actually being realized. 


PH § Another reason for emphasizing results is to encourage greater knowledge of the 
industry in which the corporation operates. Understandably, during the first cycle most of our 
work was centred on what the corporation was doing (i.e., an internal focus). During the 
Current cycle, more attention is given to comparing what the corporation is achieving in 


relation to others, resulting in a much improved basis for commenting on productivity issues 
by benchmarking, for example. 


-28 During the first cycle of special examinations, while we gained good knowledge of the 
systems and practices in place, we often did not link systems and practices and the 
achievement of corporate objectives well enough. 


.29 Figure 3 illustrates the relationship between results and systems and practices. 


s of the Authors 
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FIGURE 3: RESULTS/SYSTEMS & PRACTICES RELATIONSHIP 


RESULTS 


NOT ACHIEVED ACHIEVED 


. Misleading indicators | 1. Lack of significant S&P 
deficiencies 


. External constraints 


2. Systems and practices work 


ADEQUATE 3. Effect of new S&Ps 


not yet realized 


. Systems okay, 
practices deficient 


SYSTEMS 
AND 
PRACTICES 


5. Systems being 
ignored or bypassed 


. Deficiencies likely 
more readily 
apparent 


. Misleading indicators 


INADEQUATE 2. Results likely a product of timing 


. Likely a cause/effect 
relationship 


. Potential risks not adequately 
managed 


. Redundant or inefficient systems 
or practices 


. Good luck 


.30 


Following are a number of possible scenarios that show the relationship between 
results achieved (resources managed efficiently and economically, etc.) and the systems and 
practices employed. 


Results are being achieved and systems and practices are sound (Case B). 
Results are being achieved but systems and practices are not sound (Case D) 
Results are not being achieved and systems and practices are not sound (Case C). 


Results are not being achieved but systems and practices “appear’ to be sound 
(Case A). 
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Results-Based Auditing 


31 Figure 3 illustrates that examining systems and practices may not allow us to predict or 
conclude on results and vice versa. It also supports a more balanced approach. 


(b) What changes? 


32 By adopting more of a results-based approach to special examinations, does not mean 
Carrying out the special examination in the same way as before, but keeping results “in mind". 


33 It really means adopting a whole new approach. This can be demonstrated by stating 
the two extreme starting points: 


Systems-based approach - What are your systems and practices? 
Results-based approach - What are your objectives and are they being met? 


34 You can readily see, that the overriding question(s) you start with (i.e., audit 
objectives) will drive the whole special examination. This point is illustrated in Figure 
4. As stated eariier, it is not a matter of choosing one or the other; it is a matter of altering 
your starting point and achieving an appropriate balance. 


35 A results-oriented special examination begins with a clear understanding of the desired 
corporate results and asks what the corporation must do to meet standards of corporate 
achievement in an economical and efficient manner. 


36 In cases where the entity's objectives are unclear, auditors should identify the areas 
where greater clarity is required and consider the impact and effect of such lack of clarity on 
corporate results. Where operations and/or results are impacted in a significant manner, 
consideration should be given to reporting this as a significant deficiency. 


mY 6 In a special examination of a Crown corporation, it is impractical to examine each of 
the myriad systems and practices. Many of the organizations examined are large and 


complex. The examiner gives an opinion on the organization as a whole without looking at 
each and every aspect of the organization. 


38 The examiner must maintain a broad perspective of the organization and of the results 
it is expected to achieve, while focussing on only those areas that are critical to the 
corporation's overall success in meeting its own and FAA objectives. The plan for the 
examination must keep in sight the corporation's objectives, assess the risks against the 
achievement of those objectives and focus on systems and practices that manage the most 
important risks. The criteria for those key systems and practices would include assessment of 
the risks that need to be managed in the systems and practices. 
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Results-Based Auditing 


SYSTEMS- 


BASED 


RESULTS- 
BASED 


6. 


39 


Understanding the 
Business 


What are the 
systems and 
practices? 


What are the 
corporation's 
objectives? 

Are objectives 
being met? 
What other 
entities are 
Carrying out 
Similar activities? 
(benchmarking?) 


FIGURE 4 


Risk 
Assessment 


What are the 
major systems and 
practices? |s there 
a risk that key 
systems and 
practices will fail? 


What are key 
threats to meeting 
stated objectives? 
What are 
reasonable 
indicators of 
results being (or 
not being) 
achieved? Is 
management 
taking appropriate 
steps to deal with 
such threats/risks? 


EXAMINATION 


REPORTING 


he tance amen] | 


What should 
systems look like 
or how should 
they be 
composed? 


What are the 
expected outputs/ 
outcomes of the 
systems and 
practices? 


THE CHANGE IN VALUE-FOR-MONEY AUDITS 


Are ail the 
components of 
systems and 
practices in place 
and working? 


Are objectives being 


met, are operations 
efficient, are risks 
being managed/ 
minimized through 
the systems and 
practices 


established? Are the 


systems and 


practices sufficient to 
produce the results? 


Are there important 
deficiencies in 
systems and 
practices that can 
infer a negative 
result? Do we 
know the 
underlying reasons 
for the system 
deficiencies? 


Are there 
deficiencies in 
results and do we 
know the reasons? 
How do these 
results compare 
over time and with 
other similar 
entities or parts 
thereof? 


A results focus in value-for-money auditing is built into the audit and seen in the 


planning, the examination and the presentation of findings of the consequences or results of 
management activities. Management activities include responsibility for programs and 
systems and controls. These activities have different purposes and hence different types of 
results - for example, a service provided by a program at a reasonable price or a correct 
payment being made by a financial system. 


.40 


A results based approach can be applied to the audit of program activities and 


systems and controls. It relies on demonstrating analytically, empirically where appropriate, 


the important aspects of the results. These observations can be used in different ways to 
draw audit conclusions. 


41 


systems and controls can be presented as observations. 


1. 
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the economy or efficiency of an activity as measured against audit criteria; 


The audit observations from results focused audit tests of program activities or 
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Results-Based Auditing 


the achievement of an intermediate objective (necessary to the achievement of the 

overall program or systems and controls objective). Audit conciusions are drawn by 
comparing the observation with the intermediate objective. The achievement of the 
intermediate objective may be stated as an audit criteria. 


on the achievement of the overall objective. Audit conclusions are drawn by 
comparing the observation with the overall objective. The achievement of the overall 
objective may be stated as an audit criteria. 


42 Auditors are familiar with analytic approaches that establish causal linkages between 
observations in order to draw conclusions and make recommendations. The analytic . 
approach required in results-based auditing is distinguished by analytically demonstrating 


through audit testing and measurement where appropriate, the nature, extent and strength of 
the results. 


(a) Examples 


43 Few of the chapters in the 1992 and 1993 annual reports used a results-based 
approach extensively, but a number included some examples of its use and reporting. The 
presence of these examples demonstrates a readiness on the part of many of our auditors to 
make the shift to results-based auditing. Consistent with current practice, the examples 
focussed more on the results of management controls than on program results and were 
usually presented as observations against criteria. 


44 Some illustrative examples of the use and reporting of results from the 1992 and 1993 
annual reports are presented in Figure 2. The Figure illustrates the application of results of 
program activities and systems and controls and how the results observations were used in 
the audit. Appendix 3 provides further details from the chapters referred to in the Figure 2. 


(b) Reasons for change 
45 Many of the reasons for changing to a results-based approach in value-for-money 


audits, particularly in the examination of systems and controls, are the same as those for 
special examinations. These include: 


focussing audit effort on areas that directly affect key results of the organization being 
audited; 


providing a basis for auditing only those systems and practices essential to achieving 
key results; 


focussing audit effort on areas of importance to management; 
providing a clearer basis for assessing significance of audit findings; and 
Committees of Parliament seem to prefer discussing results rather than just systems. 


46 Value-for-money audits carried out under Section 7 of the Auditor General Act can 
have different audit objectives. For example, the objective may be to examine systems and 
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Results-Based Auditing 


controls or it may be to examine program activities. It is up to the managers of an audit to 
determine what the objectives will be. The objectives of the audit will determine how the 
results approach is used in the audit process. 


47 A results-based approach focussing on program activities has the additional advantage 
of demonstrating and potentially questioning the results that government programs have 
achieved and the difference they have made. This approach supports government and 
management concerns about challenging the vaiue of government programs as governments 
Strive to reduce budget deficits; to manage programs on results achieved rather than on 
process, and to provide improved accountability reporting on achievements. 


48 By using a results-based approach (for systems, controls and program activities) our 
reports will communicate more clearly our observations and their importance for the 
Organization being audited in terms of results. Results-based observations specify the 
importance of the observations within its own context. For example, an observation and 
conclusion will not be made simply because a "good management practice" has not been ~ 
followed (which is based on a systems based approach of what good management practice is) 
but rather because questionable results have been achieved. 


49 Auditors have always pursued the "so what" question. Rather than relying ona 
theoretical argument to demonstrate importance based on their judgment, they will rely more 
on a demonstration of results achieved. In this way they will show their willingness to test 
their own judgment, be more transparent in how they draw conclusions and more clearly make 


the case for areas that need increased attention. The effectiveness of value-for-money audits 
will, as a result, be enhanced. 


(c) What changes? 


50 The changes in the practice of value-for- money auditing are as profound as the 
changes in special examinations. 


a1 Where a value for money audit has as its objective the examination of systems and 
controls, the approach developed in the special examination applies. The only major 
difference is in the nature of the report. Special examinations, as required by legislation, 
present an overall "short-form" audit opinion with exception reporting on the deficiencies. 


Value-for-money audits provide a lengthier report that presents the details of the main 
observations and conclusions. 


52 The audit process requires auditors to make many judgments on objectives, scope, 
plans, tests, conclusions and the content of the report. A results-based approach more 
than ever demands auditors to question at each point how they will be able to test and 
demonstrate the "so what" (or the results) in their approach. The greatest challenge will 
be in the design and implementation of audit tests. These may involve new methodologies 
that may not be familiar to auditors. Methodology is being developed to assist 
auditors, and functional expertise is available in the Office. 
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Results-Based Auditing 


i CONCLUSION 


53 One of the lessons learned from the first cycle of special examinations is that a systems- 
based approach to special examinations, is a lengthy and costly endeavour which, generally, 
may not lead the audit to deal with the achievement of corporate results. A resuits-based 
approach should lead the auditor to know whether the results meet a criterion and to 
understand the most important reason for the observed performance. 


54 Many of the significant deficiencies reported five years ago consisted of weaknesses in 
systems. This was likely the result of an excessive focus on "operational elements" listed in 
Appendix 1 without linkages back to mandate and objectives or forward to results. Implicit 
was the assumption that if the systems were to be fixed, the results would improve (be better). 


ao IN a number of the special examinations completed in this current cycle, there are 
fewer of these kinds of deficiencies being reported to the Board. Matters reported are more 
directly related to the very nature of the business the corporation is in. Teams are starting to 
look outside the corporation and are considering the productivity and results of the corporation 
in relation to other similar entities or parts thereof. Indications are that we are starting to tum 
the corner on adopting a results-based approach to special examinations. 


56 lt is encouraging to see that some auditors carrying out value-for-money audits in 
departments and agencies are using results- based techniques. In many instances, however, 
the opportunity to test for and demonstrate results is not used. We thereby lose the 
opportunity to make a clearer and more convincing case of how our observations and : 
conclusions could make a significant difference. As greater pressure is placed on our audit 


resources, we will have to assure ourselves that we maximize the ability to make a difference 
through our audit investment. 


ay f By increasing our effort to test and demonstrate the results of systems, controls and 
programs, we are asking our auditors to work with techniques and approaches that they may 
not be experienced with. This presents an enormous challenge and opportunity to those 
auditors wito have used the techniques to share their experiences and to provide a helping 
Nand to other auditors trying to use them for the first time. 


58 We hope this article will encourage the shift, recognizing full well that more questions 
will arise and will need to be dealt with. 
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APPENDIX 2 
2. WHAT HAVE WE SAID? 


Special Examinations 


In Chapter 6 of the 1990 Repor, "The Audit Regime for Crown Corporations’, we 


outlined in a general way the results of our assessment of the first cycle of special 
examinations. 


A common area of weakness identified in the first special examinations related to 
unclear objectives and inadequate measurement of and reporting on performance. 
This matter was carried forward to Chapter 1 where it was stated that, "Information on 
how well Crown corporations are meeting their objectives is still largely inadequate". 


We stated that we had learned from our experience and, to ensure greater productivity 
and effectiveness in future examinations, we had initiated a number of projects 
(including, an examination of the methods used to select systems and practices for in- 
depth review, an assessment of the criteria used, etc.). 


A special examination three-day workshop was held in June 1991 at Rockland with the 
theme, "Building on Experience". As part of the session on "Understanding the Entity", 
it was stressed that more emphasis was necessary on “understanding corporate 
results". The "corporate logic model" was suggested as a tool to assist in ensuring a 
focus and understanding of mandate, objectives, inputs, activities, outputs and results 
(and linkages among them) -- see Appendix 1, "Understanding the Business" -- an 
updated version of the previous corporate logic model. 


As the "Accountability Issues" team continued with the review of the first cycle, it 
became clear that there was an excessive focus on "systems and practices" without 
adequate attention to (1) the purposes for these systems and practices, (2) whether 
the purposes were being achieved and (3) the impact of these systems and practices 
on achieving the objectives of safeguarding assets, economic and efficient 
management of resources and effectiveness of operations. This preoccupation with 
systems and process was evident throughout the process (i.e., understanding the 
entity, selecting key systems and practices for in-depth review; the wording of criteria; 
the nature and wording of identified significant deficiencies). 


We concluded that inadequate attention was being given to understanding (1) whether 
or not corporate objectives were being met and (2) the linkage of corporate systems 
and practices to the "control objectives" set out in the FAA and corporate results. 


There was Office acceptance that more of a "results" focus was needed. This issue 
became the subject of one of the series of "special examination mini workshops" held 
in January-February 1993. Many good suggestions arose, which have found their way 
into the “Special Examination Audit Guide’. At the 1993 seminar on “Planning special 
examinations, the results-based approach" was explored further, with more ideas being 
incorporated into the Audit Guide. 
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In Chapter 4 of the 1991 Report, we pointed out “Reporting Performance” as one of 
four areas of concern. We concluded: 


lf objectives are not clearly stated and the extent to which they are met is not known or 
reported, full accountability of Crown corporations cannot be achieved. 


(b) Value-for-money audits in Departments and Agencies 

One of the earlier references to the need to focus on results, was in Chapter 1 of the 

1991 Report where Denis Desautels set out some of his first impressions of Ottawa. 

Part 7 
Another impression has to do with the administration of public affairs, the need 
to focus on results instead of on process. There are cumbersome intemal rules 
in government, an emphasis on procedures, a complex budgeting process, and 
so on. To anewcomer, it seems evident that successive governments, 
concerned about probity and prudence, economy and efficiency, have placed 
pervasive systematic reliance on central controls. Why? 
One explanation is that it is hard to manage by results when few meaningful 
ways to measure them have been put into use. Reporting of results-outputs 
(efficiency) and the outcomes (effectiveness) of program activities - would 7 
appear to be a precondition for moving from a regime of central control. This is 
especially true in Ottawa. For no where in Canada is there so much talk about 
accountability which nevertheless still remains such an elusive goal. 
Further, in commenting on the deficit, one of the questions raised was: _ Has 
there been too much emphasis on process and not enough on results?" In 
introducing the idea of a deficit-related “score card", he stated, "It makes sense 
to me that the onus should now be on the government to report not only on its 
prospects for reducing the deficit, but also on the extent to which they have 
been realized. 

Finally, in that same Chapter, a section deals with, "the need to scrutinize expenditures 

in the context of outcomes". The public statements for both Crown corporations and 

departments and agencies have been consistent in encouraging public sector 

organizations to focus more on measuring and reporting results. 

in the current atmosphere that critically examines what is being accomplished by 

government programs and activities -- we see a new emphasis on results. The famous 

Liberal Red Book called for a renewed emphasis on evaluation, innovation and finding 

best practices. 

lf we want a country that works, we have to measure whether specific government 

programs actually deliver results over time. Whether it is in health care or regional 

development, we think it is important to ensure the long-term outcomes and 

consequences of our policies and programs. 

OAG Discussion Paper, August 1994 Personal views of the Authors 
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